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Why  This  Presentation 


The  role  of  the  informative  material  needs  to  be  understood 
The  role  of  the  glossary  needs  to  be  understood 
The  role  of  statistical  thinking  needs  to  be  understood 


Common  sense  is  not  so  common.  -  Voltaire 
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Evolution  of  Understanding 


Central  themes 

•  Baselines 

•  Control  Charts 

•  Statistical  management 
of  subprocesses 


Central  themes 

•  Process  Performance 
Models 

•  Understanding  and  use 
of  variation 

Supporting  themes 

•  Baselines 

•  Control  Charts 

•  Statistical  management 
of  subprocesses 
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Common  Misinterpretations 
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You  Might  Have  Misunderstood  OPP  If... 


A  table  showing  projected  defects  by  phase  looks  like  a  Process 
Performance  Model  to  you... 

The  corporate  average  “Lines  of  Code  Per  Staff  Day”  by  year  looks  like 
a  Process  Performance  Baseline  or  a  Process  Performance  Model  to 
you... 

A  control  chart  used  to  ‘manage’  defects  escaping  into  the  field  looks 
like  a  Process  Performance  Model  to  you... 

An  Earned  Value  Management  System  seems  to  fulfill  the 
requirements  of  Maturity  Level  4... 


Living  the  “High  Life” 

»  a  r.  r  ■ ■  i  .*.  .  .  m/r  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

-  Software  Engineering  Institute  CarnegieMellon  March  2008 


©2008  Carnegie  Mellon  University 


You  Might  Have  Misunderstood  QPM  If... 


“Tracking  bugs  across  the  lifecycle”  looks  like  statistical  management 
to  you... 

You  plan  to  “re-baseline”  the  control  limits  used  to  manage  critical 
subprocesses  on  a  quarterly  basis... 

‘Management  judgment’  is  used  to  ‘adjust’  control  limits  used  as 
thresholds  to  drive  corrective  actions... 

Schedule  variance  and  defect  density  look  like  perfectly  good 
subprocesses  to  statistically  manage... 
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You  Might  Have  Misunderstood  CAR  If... 


You  always  respond  to  “High  Severity”  defects  by  saying  “Let’s  run  a 
causal  analysis  and  see  what’s  going  on”... 

Causal  analysis  is  used  only  to  find  and  resolve  the  root  cause  of 
defects... 

You  don’t  see  the  value  of  applying  DAR  to  select  when  and  how  to 
apply  CAR... 

You  don’t  see  the  value  of  applying  CAR  to  select  when,  what  and  how 
to  apply  OID... 

You  don’t  see  how  Process  Performance  Models  and  Process 
Performance  Baselines  contribute  to  CAR... 
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You  Might  Have  Misunderstood  OID  If... 


You  think  42  Six  Sigma  projects  -  all  focused  on  the  inspection 
process  -  make  a  company  Maturity  Level  5... 

A  5%  boost  in  the  performance  of  a  process  that  fluctuates  by  ±7% 
looks  like  a  best  practice  to  roll  out  immediately... 

The  strength  of  an  improvement  proposal  can  only  be  measured  by  the 
persuasiveness  of  the  author... 

You  work  off  improvement  proposals  only  in  the  order  in  which  they 
were  received... 

You  don’t  see  how  Process  Performance  Models  and  Process 
Performance  Baselines  contribute  to  OID... 
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PARTI 

DEFINITIONS 
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Glossary  Use 


“The  CMMI  glossary  of  terms  is  not  a  required,  expected,  or 
informative  component  of  CMMI  models.  You  should  interpret  the 
terms  in  the  glossary  in  the  context  of  the  model  component  in  which 
they  appear”. 


"We  developed  the  glossary  recognizing  the  importance  of  using 
terminology  that  all  model  users  can  understand.  We  also  recognized 
that  words  and  terms  can  have  different  meanings  in  different  contexts 
and  environments.  The  glossary  in  CMMI  models  is  designed  to 
document  the  meanings  of  words  and  terms  that  should  have  the 
widest  use  and  understanding  by  users  of  CMMI  products." 
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Definitions  -1 


capable  process 

•  A  process  that  can  satisfy  its  specified  product  quality,  service  quality, 
and  process-performance  objectives.  (See  also  “stable  process,” 
“standard  process,”  and  “statistically  managed  process.”) 

causal  analysis 

•  The  analysis  of  defects  to  determine  their  cause, 
common  cause  of  process  variation 

•  The  variation  of  a  process  that  exists  because  of  normal  and  expected 
interactions  among  the  components  of  a  process.  (See  also  “special 
cause  of  process  variation.”) 


Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  200s  12 

©  2008  Carnegie  Mellon  University 

Definitions  -2 


establish  and  maintain 

•  In  the  CMMI  Product  Suite,  you  will  encounter  goals  and  practices  that 
include  the  phrase  “establish  and  maintain.”  This  phrase  means  more 
than  a  combination  of  its  component  terms;  it  includes  documentation 
and  usage.  For  example,  “Establish  and  maintain  an  organizational 
policy  for  planning  and  performing  the  organizational  process  focus 
process”  means  that  not  only  must  a  policy  be  formulated,  but  it  also 
must  be  documented,  and  it  must  be  used  throughout  the  organization. 
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Definitions  -3 


objectives  for  quality  and  process  performance 

•  Objectives  and  requirements  for  product  quality,  service  quality,  and 
process  performance.  Process-performance  objectives  include  quality; 
however,  to  emphasize  the  importance  of  quality  in  the  CMMI  Product 
Suite,  the  phrase  quality  and  process-performance  objectives  is  used 
rather  than  just  process-performance  objectives. 

optimizing  process 

•  A  quantitatively  managed  process  that  is  improved  based  on  an 
understanding  of  the  common  causes  of  variation  inherent  in  the 
process.  The  focus  of  an  optimizing  process  is  on  continually  improving 
the  range  of  process  performance  through  both  incremental  and 
innovative  improvements.  (See  also  “common  cause  of  process 
variation,”  “defined  process,”  and  “quantitatively  managed  process.”) 
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Definitions  -4 


process-performance 

•  A  measure  of  actual  results  achieved  by  following  a  process.  It  is 
characterized  by  both  process  measures  (e.g.,  effort,  cycle  time,  and 
defect  removal  efficiency)  and  product  measures  (e.g.,  reliability,  defect 
density,  and  response  time). 

process-performance  baselines 

•  A  documented  characterization  of  the  actual  results  achieved  by 
following  a  process,  which  is  used  as  a  benchmark  for  comparing 
actual  process  performance  against  expected  process  performance. 
(See  also  “process  performance.”) 
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Definitions  -5 


process-performance  models 

•  A  description  of  the  relationships  among  attributes  of  a  process  and  its 
work  products  that  is  developed  from  historical  process-performance 
data  and  calibrated  using  collected  process  and  product  measures  from 
the  project  and  that  is  used  to  predict  results  to  be  achieved  by 
following  a  process. 

quantitatively  managed  process 

•  A  defined  process  that  is  controlled  using  statistical  and  other 
quantitative  techniques.  The  product  quality,  service  quality,  and 
process-performance  attributes  are  measurable  and  controlled 
throughout  the  project.  (See  also  “defined  process,”  “optimizing 
process,”  and  “statistically  managed  process.”) 
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Definitions  -6 


special  cause  of  process  variation 

•  A  cause  of  a  defect  that  is  specific  to  some  transient  circumstance  and 
not  an  inherent  part  of  a  process.  (See  also  “common  cause  of  process 
variation.”) 

stable  process 

•  The  state  in  which  all  special  causes  of  process  variation  have  been 
removed  and  prevented  from  recurring  so  that  only  the  common  causes 
of  process  variation  of  the  process  remain.  (See  also  “capable 
process,”  “common  cause  of  process  variation,”  “special  cause  of 
process  variation,”  “standard  process,”  and  “statistically  managed 
process.”) 
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Definitions  -7 


statistical  process  control 

•  Statistically  based  analysis  of  a  process  and  measurements  of  process 
performance,  which  will  identify  common  and  special  causes  of 
variation  in  the  process  performance  and  maintain  process 
performance  within  limits.  (See  also  “common  cause  of  process 
variation,”  “special  cause  of  process  variation,”  and  “statistically 
managed  process.”) 

statistical  techniques 

•  An  analytic  technique  that  employs  statistical  methods  (e.g.,  statistical 
process  control,  confidence  intervals,  and  prediction  intervals). 
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Definitions  -8 


statistically  managed  process 

•  A  process  that  is  managed  by  a  statistically  based  technique  in  which 
processes  are  analyzed,  special  causes  of  process  variation  are 
identified,  and  performance  is  contained  within  well-defined  limits.  (See 
also  “capable  process,”  “special  cause  of  process  variation,”  “stable 
process,”  “standard  process,”  and  “statistical  process  control.”) 
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PART  II 

FUNDAMENTALS  OF 
STATISTICAL  THINKING 
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Fundamentals  of  Statistical  Thinking 


All  product  development  and  services  are  a  series  of  interconnected 
processes. 

All  processes  have  variation  in  their  results. 

Understanding  variation  is  the  basis  for  management  by  fact  and 
systematic  improvement: 

•  the  past  quantitatively 

•  the  present  quantitatively 

•  the  future  quantitatively 
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What  Is  a  Process  in  Relation  to  Products  and 


Processes  defined  in  CMMI  are  “activities  that  can  be  recognized  as 
implementations  of  practices  in  a  CMMI  model.” 

They  may  also  be  thought  of  as  a  system  that  includes  the  people, 
materials,  energy,  equipment,  and  procedures  necessary  to  produce  a 
product  or  service. 


Requirements 
&  Ideas 


People 


Material  Energy  Equipment  Procedures 
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Distributions  Describe  Variation 


Populations  of  data  are  characterized  as  distributions  in  most  statistical 
procedures: 

•  expressed  as  an  assumption  for  the  procedure 

•  can  be  represented  using  an  equation 


The  following  are  examples  of  distributions  you  may  come  across: 
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How  Distributions  Are  Formed 
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What  Is  a  Statistic? 


A  summary  or  characterization  of  a  distribution  (i.e.,  a  set  of  numbers) 


A  characterization  of  a  central  tendency  (e.g.,  mean,  median,  and 
mode) 

A  characterization  of  Useful  Probabilities  for  Normal  Dis 

_  6  8  %  _ 

dispersion  (e.g.,  variance,  j  1  95o/° _ j _ 

standard  deviation,  j - 1 - f —  99%  — - 

interquartile  range,  and  range) 


Central  Tendency  and  Dispersion 


Central  tendency  implies  location: 

•  middle  of  a  group  of  values 

•  balance  point 

•  examples  include  mean,  median,  and  mode 
Dispersion  implies  spread: 

•  distance  between  values 

•  how  much  the  values  tend  to  differ  from  one  another 

•  examples  include  range  and  (sample)  standard  deviation 

These  two  are  used  together  to  understand  the  baseline  of  a  process- 
performance  factor  and  outcome. 
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Other  Terms  and  Definitions 


A  consists  of  the 

total  possible  observations  with 
which  you  are  concerned  but  to 
which  you  do  not  necessarily 
have  access  (Xi  thru  X15 


A 

is  a  set  of  observations 
selected  from  a  population 
that  you  can  access. 


(x2) 


X|> 


Statistics  (specifically  hypothesis  testing)  enable  you  to  place  a 
confidence  interval  on  the  central  tendency  and  variation  of  the 
population  and  on  future  samples. 
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Hypothesis  Testing:  To  Understand  and 
Compare  Performance 


A  formal  way  of  making  a  comparison  and  deciding  whether  or  not  the 
difference  is  significant  is  based  on  statistical  analysis. 

Hypothesis  testing  consists  of  a  null  and  alternative  hypothesis: 

•  The  states  that  the  members  of  the  comparison 

are  equal;  (a  concrete,  default  position). 


•  The  alternative  hypothesis  states  that  there  is  a  difference;  it  is 

supported  when  the  null  hypothesis  is  rejected. 


The  conclusion  either  rejects  or  fails  to  reject  the  null  hypothesis. 


Understanding  the  null  and  alternative  hypotheses  is  the 
key  to  understanding  the  results  of  statistical  prediction 
models  discussed  in  Module  5  on  OPP. 
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Formally  Stating  a  Hypothesis 


Average  productivity  equals  100  source  lines  of  code  (SLOC)  per  person 
week: 


•  Null:  Average  productivity  is  equal  to  100  SLOC  per  person  week. 

•  Alternative:  Average  productivity  is  not  equal  to  100  SLOC  per 
person  week. 

A  refinement  of  these  hypotheses  are  as  follows: 

•  Null:  Average  productivity  is  equal  to  100  SLOC  per  person  week. 

•  Alternative:  Average  productivity  is  less  than  100  SLOC  per  person 
week. 

Generally,  the  alternative  hypothesis  is  the  difference  (e.g.  improvement  or 
performance  problem)  that  you  seek  to  learn  about. 


The  null  hypothesis  holds  the  conservative  position  that  apparent 
differences  can  be  explained  by  chance  alone.  The  phrase  “is  equal  to” 
will  generally  appear  in  the  null  hypothesis. 
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Slogan  to  Remember  p  Interpretation 


V 


■\ 


When  the  p  is  low,  the  null  must  go. 
When  the  p  is  high,  the  null  must  fly. 


Note:  The  p  value  is  the  key  output  in  statistical  analysis  that  students  are  taught 
to  identify  and  use  to  draw  a  conclusion  regarding  the  hypothesis  test 
comparison  or  regarding  the  significance  of  a  statistical  model. 
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Data  Type  Determines  the  Hypothesis  Test 


Attribute  Nominal 

(aka  categorized 
discrete  data) 


Continuous 

(aka  variables  data) 


Ordinal 


Interval 

Ratio 


Categorical  data  where  the  order  of  the 
categories  is  arbitrary  _ 

wL&S 

A  B  C 

Nominal  data  with  an  ordering; 
may  have  unequal  intervals 


Examples 
Defect  types 
Labor  types 
Languages 

Examples 
Severity  levels 
Survey  choices  1-5 
Experience  categories 


Continuous  data  that  has  equal 
intervals;  may  have  decimal  values 

Interval  data  set 
that  also  has 
a  true  zero  point; 
decimal  values 


Examples 
Defect  densities 
Labor  rates 
Productivity 
Variance  %’s 
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Prediction  Modeling  Techniques 

Y 


Continuous  Discrete 


ANOVA 

&  Dummy  Variable 
Regression 

Chi-Square 
&  Logistic  Regression 

Correlation 
&  Regression 

Logistic  Regression 
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p  value  Summary 


Method  Null  Alternative  P  <  0.05  P  >  0.05 


Hypothesis 

Tests 

No  difference  exists; 
no  associations 

Two  items  are  different; 
association  exists 

Accept 

alternative 

Accept  null 

Tests  for 
Normality 

Data  follows  Normal 
Distribution 

Data  does  not  follow 
Normal  Distribution 

Accept 

alternative 

Accept  null 

AN  OVA 

No  difference  of  Y 
across  levels  of  x 

Difference  of  Y  exists 
between  1+  levels  of  x 

Accept 

alternative 

Accept  null 

Regression 

x  factor  does  not  add 
value  to  model 

X  factor  adds  value  to 
model 

Accept 

alternative 

Accept  null 

Chi-Square 

Two  discrete 
variables  are  not 
associated 

Two  discrete  variables 
are  associated 

Accept 

alternative 

Accept  null 

Logistic 

Regression 

x  factor  does  not  add 
value;  model  has  no 
significant  x’s 

X  factor  adds  value  to 
model;  model  has  In¬ 
significant  x’s 

Accept 

alternative 

Accept  null 
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PART  III 

A  TALE  OF  TWO 
ORGANIZATIONS 
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Introduction 


The  tale  of  two  organizations  aspiring  for  CMMI  High  Maturity  is 
embedded  in  the  next  section 

The  first  organization,  called  “ Un-Gestalt ,  does  not  view  the  CMMI 
holistically,  nor  use  the  informative  material  to  guide  practice. 

The  second  organization,  called  “ Gestalt ,  wants  to  use  the  CMMI  High 
Maturity  practices,  including  informative  material,  to  gain  true  competitive 
advantage  and  grow  their  business. 
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Caveats 


The  tale  demonstrates  differences  in  practical  use  and  benefit  of  CMMI 
High  Maturity  Practices 

The  “Un-Gestalf  organization  thinks  they  are  performing  acceptably  at 
the  CMMI  High  Maturity  level  but  in  fact  are  not 

The  “Gestalf  organization  epitomizes  an  exemplary  interpretation  and 
implementation  of  CMMI  High  Maturity  practices. 

The  tale  illustrates  the  importance  of  understanding  variation  in 
addition  to  central  tendency,  the  benefit  of  having  reliable  knowledge  of 
causal  relationships  beyond  trends,  and  the  benefit  of  having  finer- 
grained  insight  into  process  performance,  as  contrasted  with  less 
frequent  and  larger-grained  monitoring  of  process  performance. 

The  “Gestalt”  example  illustrates  superior  methods  within  the 
mainstream  of  industry  use;  however,  the  “Gestalt”  example  is  not  a 
prescription  for  what  is  a  minimal  acceptable  interpretation  from  either 
a  maturity  rating  or  an  appraisal  perspective. 
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References  for  the  Gestalt  Examples 


•  http://www.isixsiqma.com 


•  http://www.allbusiness.com 

Query  on  the  following  terms  and  “Case  Study”: 


ANOVA 
Chi-Square 
Regression 
Logistic  Regression 
Dummy  Variable  Regression 
Bayesian  Belief  Network 
Designed  Experiments 
Discrete  Event  Simulation 


Reliability  Growth  Modeling 
Response  Surface  Modeling 
Time  Series  Analysis 
Hypothesis  Testing 
Logit 

Monte  Carlo  Simulation 
Optimization 
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Recent  Publications 


•  “CMMI  and  Six  Sigma:  Partners  in  Process  Improvement”  by 
Jeannine  M.  Siviy,  M.  Lynn  Penn,  and  Robert  W.  Stoddard. 
Addison-Wesley  2008. 

•  “Moving  Up  the  CMMI  Capability  and  Maturity  Levels  Using 
Simulation”  by  David  M.  Raffo,  PhD  and  Wayne  Wakeland,  PhD. 
CMU/SEI-2008-TR-002,  January  2008. 

(http://www.sei.cmu.edu/publications/documents/08.reports/08tr002. 

html) 
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Scenarios  within  the  Tale 


1.  Establishing  Process  Performance  Baselines  (PPB) 

2.  Deciding  on  Process  Performance  Models  (PPM) 

3.  Project  Forecasting  (PM) 

4.  Composing  a  Process  (Compose) 

5.  Deciding  What  to  Statistically  Manage  (Manage) 

6.  Periodic  Management  Reviews  of  Projects  (Reviews) 

7.  Taking  Corrective  Action  When  Needed  (CAR) 

8.  Introducing  Innovative  Change  to  Organization  (OID) 
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PART  IV 

LEVELS  4  AND  5- 
TO  "GESTALT," 

NOT  “UN-GESTALT” 
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MDD  on  Use  of  Informative  Material  and 
Subpractices  -1 


The  MDD  states  on  page  1-20 

•  "Appraisal  teams  compare  the  objective  evidence  collected 
against  the  corresponding  practices  in  the  appraisal  reference 
model.  In  making  inferences  about  the  extent  to  which  practices 
are  or  are  not  implemented,  appraisal  teams  draw  on  the  entire 
model  document  to  understand  the  intent  of  the  model,  and  use 
it  as  the  basis  for  their  decisions.  This  comparison  includes  the 
required  and  expected  model  components  (i.e.,  generic  and 
specific  goals,  generic  and  specific  practices)  as  well  as 
informative  material,  such  as  model  front  matter,  introductory 
text,  glossary  definitions,  and  subpractices." 
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MDD  on  Use  of  Informative  Material  and 
Subpractices  -2 


Additionally  on  page  1-24  in  discussing  direct  artifacts  for  Plls 

•  "The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  May  be  explicitly  stated  or  implied  by  the 
practice  statement  or  associated  informative  material." 

And  from  page  11-110 

•  "The  use  of  informative  material  in  the  appraisal  reference 
model  to  form  a  checklist  is  explicitly  discouraged." 

And  from  page  111-50  the  glossary  definition  for  direct  artifact 

•  “The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  May  be  explicitly  stated  or  implied  by  the 
practice  statement  or  associated  informative  material. " 
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Interpreting  this  Presentation 


i  Text  in  the  yellow  boxes  is  an  example  description  of  implementing  | 
|  the  practice  consistent  with  the  glossary,  using  the  standard  English  ! 
;  meaning  of  words  instead  of  the  statistical  meaning,  and  without 
I  using  the  informative  material.  For  example,  interpreting  variation  to  \ 
mean  the  difference  between  two  items.  ! 

i _ j 


Text  in  the  green  boxes  is  an  example  description  of  implementing 
the  practice  consistent  with  the  glossary,  the  statistical  meaning  of 
words,  and  accounting  for  the  informative  material.  For  example, 
interpretating  variation  (in  the  level  4  &  5  practices)  to  mean  central 
tendency  and  dispersion. 
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OPP  SG  1  Establish  Performance  Baselines 
and  Models 


Baselines  and  models,  which  characterize  the  expected  process 
performance  of  the  organization's  set  of  standard  processes,  are 
established  and  maintained. 
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OPP  SP  1.1  Select  Processes 


Select  the  processes  or  subprocesses  in  the  organization’s  set  of 
standard  processes  that  are  to  be  included  in  the  organization’s 
process-performance  analyses. 

|  Pick  a  few  processes  from  the  OSSP  for  which  we  have  measures,  j 


Select  processes/subprocesses  that  will  help  us  understand  our 
ability  to  meet  the  objectives  of  the  organization  and  projects,  and  the 
need  to  understand  quality  and  process  performance.  These 
subprocesses  will  typically  be  the  major  contributors  and/or  their 
measures  will  be  the  leading  indicators. _ 
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OPP  SP  1.2  Establish  Process-Performance 
Measures 


Establish  and  maintain  definitions  of  the  measures  that  are  to  be 
included  in  the  organization’s  process-performance  analyses. 


I  Provide  definitions  for  the  measures  and  update  as  necessary. 

i _ 


i 


i 


Select  measures,  analyses,  and  procedures  that  provide  insight  into 
the  organization’s  ability  to  meet  its  objectives  and  into  the 
organization’s  quality  and  process  performance.  Create/update  clear 
unambiguous  operational  definitions  for  the  selected  measures. 
Revise  and  update  the  set  of  measures,  analyses,  and  procedures  as 
warranted.  In  usage,  be  sensitive  to  measurement  error.  The  set  of 
measures  may  provide  coverage  of  the  entire  lifecycle  and  be 
controllable. 
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OPP  SP  1.3  Establish  Quality  and  Process- 
Performance  Objectives 


Establish  and  maintain  quantitative  objectives  for  quality  and  process 
performance  for  the  organization. 


Write  down  quality  and  process  performance  objectives  such  as 
improve  cycle  time,  quality,  and  the  percent  of  improvement  we  want. 


These  objectives  will  be  derived  from  the  organization’s  business 
objectives  and  will  typically  be  specific  to  the  organization,  group,  or 
function.  These  objectives  will  take  into  account  what  is  realistically 
achievable  based  upon  a  quantitative  understanding  (knowledge  of 
variation)  of  the  organization’s  historic  quality  and  process 
performance.  Typically  they  will  be  SMART  and  revised  as  needed. 
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OPP  SP  1.4  Establish  Process-Performance 
Baselines 


Establish  and  maintain  the  organization's  process-performance 
baselines. 

Store  measures  in  our  spreadsheet  repository  on  a  periodic  basis 
indicating  the  end  date  of  the  period  they  represent  and  baseline 
them  in  our  CM  system. 

Baselines  will  be  established  by  analyzing  the  distribution  of  the  data 
to  establish  the  central  tendency  and  dispersion  that  characterize  the 
expected  performance  and  variation  for  the  selected 
process/subprocess.  These  baselines  may  be  established  for  single 
processes,  for  a  sequence  of  processes,  etc.  When  baselines  are 
created  based  on  data  from  unstable  processes,  it  should  be  clearly 
documented  so  the  consumers  of  the  data  will  have  insight  into  the 
risk  of  using  the  baseline.  Tailoring  may  affect  comparability  between 
baselines. 


Living  the  “High  Life” 

r.  i-  ■  ■  ,  ■ ,  .  i 7  .  ■  ri  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

-  Software  Engineering  Institute  CarnegieMellon  March  2008 

©  2008  Carnegie  Mellon  University 


Scenario  1:  Establishing 
Process  Performance 
Baselines 
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Scenario  1  (PPB):  “Un-Gestalt” 


We  have  performance  baselines  on  a  variety  of  factors.  For  example, 
we  know  that  we  have  the  following  average  defect  density  (defects 
per  10  KSLOC)  entering  System  Test: 

•  14.35  algorithm  defects 

•  1 3.20  stack  overflow  defects 

We  focused  most  of  our  effort  on  the  algorithm  defects  using  pareto 
analysis,  not  realizing  ... 
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Scenario  1  (PPB):  “Un-Gestalt”  -  continued 


Two-sample  T  for  Algorithm  vs  StackOverFlow 


Algorithm 

StackOverFlow 


M 

Mean 

StDev 

SE  Mean 

100 

14.35 

6.07 

0.61 

100 

13.2  0 

4.53 

0.45 

(Algorithm) 

-  mu 

(StackOverFlow) 

Estimate  for  difference:  1.154 

95%  Cl  for  difference:  (-0.340,  2.649) 

T-Test  of  difference  =  0  (vs  not  =} :  T-Valne  =  1.52 


P -Value  =  0.129 


The  P-Value  greater  than  0.05  shows  that  we  cannot  reject  the  Null 
Hypothesis  (that  these  two  defect  types  occur  at  similar  rates)! 

Thus,  we  should  be  focusing  on  both  types  of  defects  equally! 
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Scenario  1  (PPB):  “Gestalt” 


We  have  performance  baselines  on  a  variety  of  factors.  For  example,  we 
know  from  last  year  that  we  have  the  following  baselines  which  follow  the 
normal  distribution: 


Detect  lype 
Entering  Test 

Mean 

Std  Dev 

Algorithm 

15 

2.5 

Stack  Overflow 

10 

3.3 

Global  Variables 

7 

1.98 

Processing  Logic 

5 

0.76 

Data  Type  Mismatch 

5 

0.23 

Invalid  Pointers 

3 

0.12 

Cosmetic 

9 

1.98 

Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  ii  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  200s  52 

©  2008  Carnegie  Mellon  University 

Scenario  1  (PPB):  “Gestalt”  -  continued 


Knowing  the  distribution  of  each  performance  baseline,  we  are  able  to 
confidently  assess  whether  we  have  real  “differences”  to  act  upon  or  not. 

We  use  ANOVA  to  assess  true  differences! 
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Scenario  1  (PPB):  “Gestalt”  -  continued 


One-way  ANOVA:  Algorithm,  StackOverFlow,  GlobalVariables,  ProcessingLogic,  DataTypeMismatch,  InvalidPointers,  Cosmetic 


Source 

DF 

SS 

MS 

F 

Factor 

6 

11189.30 

1864 .88 

198.56 

Error 

693 

6508 . 68 

9.39 

Total 

699 

17697 . 98 

0.000 


Level 

N 

Mean 

StDo^r 

Algorithm 

100 

14.354 

6.0/7 

StackOverFlow 

100 

13.200 

4 . 334 

GlobalVariables 

100 

6.927 

1.169 

ProcessingLogic 

100 

5.158 

0.167 

DataTypeMismatch 

100 

4 . 971 

0.^30 

InvalidPointers 

100 

2 . 994 

0.1^ 

Cosmetic 

100 

9.037 

1.96\ 

S  =  3.065  R-Sq  =  63.22%  R-Sq(adj)  =  62. 


^Individual  95%  CIs  For  Mean  Based  on 
Pooled  StDev 

(-*-) 

(-*) 

(-*-) 

(-*) 

(-*-) 

(-*) 

(-*-) 

5  7.0  10.5  14.0 
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OPP  SP  1.5  Establish  Process-Performance 
Models 


Establish  and  maintain  the  process-performance  models  for  the 
organization’s  set  of  standard  processes. 

We  have  historical  productivity  and  defect  injection/detection  rates  by 
phase  which  we  update  periodically  and  include  in  reports. 

Rather  than  just  a  point  estimate,  PPMs  will  address  variation  in  the 
prediction.  PPMs  will  model  the  interrelationships  between 
subprocesses  including  controllable/uncontrollable  factors.  They 
enable  predicting  the  effects  on  downstream  processes  based  on 
current  results.  They  enable  modeling  of  a  PDP  to  predict  if  the 
project  can  meet  its  objectives  and  evaluate  various  alternative  PDP 
compositions.  They  can  predict  the  effects  of  corrective  actions  and 
process  changes.  They  can  also  be  used  to  evaluate  the  effects  of 
new  processes  and  technologies/innovations  in  the  OSSP. 
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Scenario  2:  Deciding  on 
Process  Performance 
Models 
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Scenario  2  (PPM):  “Un-Gestalt” 


We  are  using  both  COCOMO  and  SLIM  for  our  initial  project 
forecasting.  These  models  have  predictive  value  and  may  be  used  by 
answering  a  list  of  questions. 

We  do  our  very  best  with  these  models  to  give  them  the  best  starting 
point  as  possible. 

We  also  have  an  escaped  defect  model  that  uses  the  historical 
average  defects  inherited,  injected  and  removed  by  phase. 

Even  with  these  models,  we  still  seem  to  have  plenty  of  surprises  in 
cost,  schedule  and  quality! 
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Scenario  2  (PPM):  “Gestalt” 


We  have  enriched  our  detailed  process  maps  from  CMMI  ML3  to 
include  executable  process  models  that  possess  information  on  cycle 
times,  processing  times,  available  resources,  sub-process  costs  and 
quality. 

We  have  also  identified  the  key  process  handoffs  during  the  project 
execution  in  which  exit  and  entrance  criteria  are  important! 

At  these  handoffs,  we  have  process  performance  models  predicting  the 
interim  outcomes.  They  will  form  a  pact  governing  the  process  handoff 
and  provide  leading  indicators  of  problems  with  outcomes. 
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Scenario  2  (PPM):  “Gestalt” 


An  illustration  of  an 
appropriate  number  of  PPMs. 

In  Swimming,  the  three 
primary  subprocesses  are  1) 
entering  the  water,  2) 
straightline  swim,  and  3) 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


Next,  we  identify  controllable  factors  tied  to  earlier  sub-processes  that 
may  be  predictive  of  one  or  more  of  the  outcomes  (interim  and  final) 
we  need  to  predict. 


We  then  decide  what  type  of  data  our  outcome  (Y)  is  and  what  type  of 
data  our  factors  (x’s)  are. 


Using  the  data  types,  we  can  then  begin  to  identify  the  statistical 
methods  to  help  with  our  modeling.  (See  next  slide) 
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Continuous  Discrete 


Scenario  2  (PPM):  “Gestalt”  -  continued 


Y 


Continuous  Discrete 


AN  OVA 
&  MAN OVA 

&  Dummy  Variable 
Regression 

Chi-Square 
&  Logit 

Correlation 
&  Regression 

Logistic  Regression 

Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  2008  61 

©  2008  Carnegie  Mellon  University 

Scenario  2  (PPM):  “Gestalt”  -  continued 


ANOVA,  Dummy  Variable  Regression 


Using  these  controllable  factors... 

To  predict  this  outcome! 

Type  of  Reviews  Conducted;  Type  of  Design 
Method;  Language  Chosen;  Types  of  Testing 

Delivered  Defect  Density 

High-Medium-Low  Domain  Experience; 

Architecture  Layer;  Feature;  Team;  Lifecycle 
model;  Primary  communication  method 

Productivity 

Estimation  method  employed;  Estimator;  Type  of 
Project;  High-Medium-Low  Staff  Turnover;  High- 
Medium-Low  Complexity;  Customer;  Product 

Cost  and  Schedule 

Variance 

Team;  Product;  High-Medium-Low  Maturity  of 
Platform;  Maturity  or  Capability  Level  of  Process; 
Decision-making  level  in  organization;  Release 

Cycle  Time  or 
Time-to-Market 

Iterations  on  Req’ts;  Yes/No  Prototype;  Method  of 
Req’ts  Elicitation;  Yes/No  Beta  Test;  Yes/No  On- 
Time;  High-Medium-Low  Customer  Relationship 

Customer  Satisfaction  (as 
a  percentile  result) 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


Regression 


Using  these  controllable  factors... 

To  predict  this 
outcome! 

Req’ts  Volatility;  Design  and  Code  Complexity; 
Test  Coverage;  Escaped  Defect  Rates 

Delivered  Defect  Density 

Staff  Turnover  %;  Years  of  Domain  Experience; 
Employee  Morale  Survey  %;  Volume  of 
Interruptions  or  Task  Switching 

Productivity 

Availability  of  Test  Equipment  %;  Req’ts 

Volatility;  Complexity;  Staff  Turnover  Rates 

Cost  and  Schedule 
Variance 

Individual  task  durations  in  hrs;  Staff  availability 
%;  Percentage  of  specs  undefined;  Defect 
arrival  rates  during  inspections  or  testing 

Cycle  Time  or 
Time-to-Market 

Resolution  time  of  customer  inquiries; 

Resolution  time  of  customer  fixes;  Percent  of 
features  delivered  on-time;  Face  time  per  week 

Customer  Satisfaction 
(as  a  percentile  result) 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


Chi-Square,  Logistic  Regression 


Using  these  controllable  factors... 

To  predict  this  outcome! 

Programming  Language;  High-Medium-Low 
Schedule  compression;  Req’ts  method;  Design 
method;  Coding  method;  Peer  Review  method 

Types  of  Defects 

Predicted  Types  of  Defects;  High-Medium-Low 
Schedule  compression;  Types  of  Features 
Implemented;  Parts  of  Architecture  Modified 

Types  of  Testing  Most 
Needed 

Architecture  Layers  or  components  to  be 
modified;  Type  of  Product;  Development 
Environment  chosen;  Types  of  Features 

Types  of  Skills  Needed 

Types  of  Customer  engagements;  Type  of 
Customer;  Product  involved;  Culture;  Region 

Results  of  Multiple  Choice 
Customer  Surveys 

Product;  Lifecycle  Model  Chosen;  High-Medium- 
Low  Schedule  compression;  Previous  High  Risk 
Categories 

Risk  Categories  of  Highest 
Concern 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


Logistic  Regression 


Using  these  controllable  factors... 

To  predict  this 
outcome! 

Inspection  Preparation  Rates;  Inspection  Review 
Rates;  Test  Case  Coverage  %;  Staff  Turnover 

Rates;  Previous  Escape  Defect  Rates 

Types  of  Defects 

Escape  Defect  Rates;  Predicted  Defect  Density 
entering  test;  Available  Test  Staff  Hours;  Test 
Equipment  or  Test  Software  Availability 

Types  of  Testing  Most 
Needed 

Defect  Rates  in  the  Field;  Defect  rates  in  previous 
release  or  product;  Turnover  Rates;  Complexity  of 
Issues  Expected  or  Actual 

Types  of  Skills  Needed 

Time  (in  Hours)  spent  with  Customers;  Defect 
rates  of  products  or  releases;  Response  times 

Results  of  Multiple  Choice 
Customer  Surveys 

Defect  densities  during  inspections  and  test;  Time 
to  execute  tasks  normalized  to  work  product  size 

Risk  Categories  of 

Highest  Concern 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


Recently,  we  conducted  a  regression  analysis  to  develop  our 
statistically-based  process  performance  model  predicting  Defect 
Density. 


As  will  be  seen  on  the  next  slide,  the  regression  model  provides  rich 
information  about  the  role  of  the  controllable  x  factors  (Req’ts 
Volatility  and  Experience)  in  predicting  the  Y  outcome  (Defect 
Density). 


In  turn,  this  will  provide  management  with  rich  information  on  how  to  be 
pro-active  in  changing  predicted  high  levels  of  Defect  Density  to 
acceptable  lower  levels! 
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Scenario  2  (PPM):  “Gestalt”  -  continued 

Regression  Analysis:  Defect  Densi  versus  ReqtsVolatil.  YearsDomainE 


5  recession  equation  i3 

Dpfer.t.  Dens  i  ty  =  0 . 4  8  4  +  0.480  Reqt 3V0  la  t  i  i  i  ty  -  0.0242  Yea  r s  Dorna  i  nE  xpe  r  iencn 


Er edict or 

Constant 

Reqt  sVo 1 a  t i 1 i  ty 

Yea  r  s  Dorna i nE  xperie nee 


Coef  SE  Coef 


0.48357 

0.47953 

■0.024215 


0.03957 

0.09511 

0.001941 


1 

12.22 

5.04 


E 

0  .  I J  M  o 

0.000 


Prediction  equation 
of  defect  density 


■12.48  \£L.  000/  ^ 


|S  =  0.00893207  R-5q  =  85.9% 


Analysis  of  Variance 


Source 
Regression 
Residual  Err; 
Total 


DF 


p  value  below  0.05 
indicates  the  model  is 
significant 


p  values  below  0.05 
indicate  the 
predictors  to  keep  in 
^the  model _ 
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Percentage  of  total 
variation  in  defect 
density  explained  by 
the  model 
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Scenario  2  (PPM):  “Gestalt”  -  continued 


A  probabilistic  model  can  represent  a  collection  of  process 

performance  models  in  that  each  child  node  below  may  be  statistically 
predicted  by  it’s  parents  to  the  left. 


Req’ts  Architecture  Design  Code  Test  Release 


OPP  SG  1  Establish  Performance  Baselines 
and  Models 


Baselines  and  models,  which  characterize  the  expected  process 
performance  of  the  organization's  set  of  standard  processes,  are 
established  and  maintained. 

r - - - 

I  The  aforementioned  data  and  models  characterize  OSSP 
!  performance. 


Central  tendency  and  variation  are  the  cornerstones  of  our 
implementation.  Our  baselines  and  models  incorporate  our 
understanding  of  these,  allow  us  to  understand  risks  in  our 
organizations  and  its  projects,  and  allow  us  to  create  and  execute 
effective  strategies  to  mitigate  and  manage  risks. 
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QPM  SG  1  Quantitatively  Manage  the  Project 


The  project  is  quantitatively  managed  using  quality  and  process- 
performance  objectives. 
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QPM  SP  1.1  Establish  the  Project’s  Objectives 


Establish  and  maintain  the  project’s  quality  and  process-performance 
objectives. 

r - 

!  Project  Manager  documents  project  objectives  such  as  “Produce  the 

^system  better,  cheaper,  faster”  in  the  project  plan. 


These  objectives  will  be  based  on  the  organization’s  quality  and 
process  performance  objectives  and  any  additional  customer  and 
relevant  stakeholder  needs  and  objectives.  These  objectives  will  be 
realistic  (based  upon  analysis  of  historical  quality  and  process 
performance)  and  will  cover  interim,  supplier,  and  end-state 
objectives.  Conflicts  between  objectives  (i.e.,  trade-offs  between 
cost,  quality,  and  time-to-market)  will  be  resolved  with  relevant 
stakeholders.  Typically  they  will  be  SMART,  traceable  to  their  source, 
and  revised  as  needed. 
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QPM  SP  1.2  Compose  the  Defined  Process 


Select  the  subprocesses  that  compose  the  project’s  defined  process 
based  on  historical  stability  and  capability  data. 

Look  at  our  data  spreadsheets  to  select  the  subprocesses  that  have 
the  highest  performance,  best  quality,  and  most  stability  --  the  ones 
that  have  changed  the  least. 


The  PDP  is  composed  by: 

•  selecting  subprocesses 

•  adjusting/trading-off  the  level  and  depth  of  intensity  of 
application  of  the  subprocess(es)  and/or  resources 

to  best  meet  the  quality  and  process  performance  objectives.  This 
can  be  accomplished  by  modeling/simulating  the  candidate  PDP(s)  to 
predict  if  they  will  achieve  the  objectives,  and  the  confidence  level  of 
(or  risk  of  not)  achieving  the  objective. 
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Scenario  3:  Project 
Forecasting 
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Scenario  3  (PM):  “Un-Gestalt” 


We  collect  data  on  historical  projects  and  use  it  to  compare  our 
projects  being  planned  to  similar  historical  projects. 

We  also  ask  each  sub-process  owner  for  their  assessment  of  task 
duration  and  we  compute  our  critical  path. 

Regretfully,  our  schedule  variances  are  not  improving  over  the  past  4 
years.  It  seems  that  we  may  have  hit  a  ceiling  of  performance  in  our 
schedule  variance! 
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Scenario  3  (PM):  “Gestalt” 


We  collect  data  on  historical  projects  and  develop  distributions  of  task 
durations  for  key  sub-processes. 

When  we  don’t  have  solid  historical  data,  we  query  the  process  owners 
for  task  durations  by  asking  them  for  fBest  Case,  Worst  Case,  Most 
Likelvl  so  that  we  can  model  the  uncertainty. 

We  have  much  fewer  surprises  in  our  schedules  with  this  approach! 
Instead  of  reporting  single  values  that  management  wants  to  hear, 
process  owners  are  honest!  Everyone  now  has  buy-in  to  the  schedule! 
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Scenario  3  (PM):  “Gestalt”  -  continued 
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Scenario  3  (PM):  “Gestalt”  -  continued 


Process 

Durations 

Step 

Best 

Expected 

Worst 

1 

27 

30 

75 

2 

45 

50 

125 

3 

72 

80 

200 

4 

45 

50 

125 

5 

81 

90 

225 

6 

23 

25 

63 

7 

32 

35 

88 

8 

41 

45 

113 

( Would  you  change  ^ 
your  mind  in  the 
—  face  of  unbalanced 

9 

63 

70 

175 

10 

23 

25 

63^^^ 

500 

— ^nsk?  J 
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Scenario  3  (PM):  “Gestalt”  -  continued 


.JSJxJ 

Edit  View  Forecast  Preferences  Help 

1.000,000  Trials 

Frequency  View 

995,599  Displayed 

Total  Duration 


^Almost 
guaranteed  to 
miss  the  500 
days  duration 
1 00%  of  the 
Vtime! 


560.00  580.00  600.00  620.00  640.00 


With  90% 
confidence,  we  will 
be  under  817  days 
duration! 


^  [-infinity 


0.00  700.00  720.00  740.00  760.00  780.00  800.00 

Days 

Certainty:  |90.0960 


.00  840.00  860.00 


4  [Ha- 
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Scenario  4:  Composing  a 
Process 


Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  200s  79 

©  2008  Carnegie  Mellon  University 

Scenario  4  (Compose):  “Un-Gestalt” 


We  know  how  our  processes  work.  We  don’t  have  a  lot  of  choices  but 
our  experts  are  confident  that  we  do  make  the  correct  few  choices 
during  our  tailoring  session. 

If  our  experts  believe  that  there  were  problems  during  the  last  project 
with  some  of  our  sub-processes,  we  may  choose  alternative  sub¬ 
processes  to  avoid  problems. 

We  believe  we  are  informed,  but  we  aren’t  always  confident  in  our 
choices  -  as  we  continue  to  have  surprises  in  process  performance! 
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Scenario  4  (Compose):  “Gestalt” 


We  have  collected  plenty  of  distributional  data  for  performance 
baselines  of  our  key  sub-processes. 

By  analyzing  our  organizational  goals  and  customer  reqts,  we  can 
model  our  subprocess’  capabilities  to  see  if  they  provide  desirable 
outcomes  in  cost,  schedule  and  quality. 

We  also  reach  into  our  process  performance  models  to  see  if  they  are 
predicting  successful  outcomes  based  our  composition  decisions. 
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Scenario  4  (Compose):  “Gestalt”  -  continued 


Our  modeling  for  process  composition  is  based  on  Monte  Carlo 
simulation  and  optimization. 

Essentially,  we  can  model  the  inter-connected  subprocesses  and 
include  decisions  of  which  alternative  subprocesses  to  choose. 

The  simulation  and  optimization  help  to  confirm  which  choices  we 
should  make. 

We  are  thankful  that  this  modeling  is  available  because  we  have  many 
complicated  processes  involving  many  tradeoffs! 
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(  Crystal  Ball  then^i 
allows  the  user  to 
analyze  and 
interpret  the  final 
^distribution  of  C! 


Crystal  Ball  uses  a 
random  number 
generator  to  select 
values  for  A  and  B 
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Crystal  Ball 
causes  Excel  to 
recalculate  all 
cells,  and  then  it 
saves  off  the 
different  results 
forC! 
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Scenario  4  (Compose):  “Gestalt”  -  continued 


Option  1  Option  2  Option  3  Option  4 


Requirements  ^ 

_  ,  x  Effort 

Development 

Cycle  Time 

Quality 

Traditional 

KJ  Analysis  &  QFD 

Prototyping 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

25 

35 

45 

35 

45 

55 

65 

80 

95 

15 

20 

25 

30 

35 

40 

50 

60 

70 

35 

45 

55 

27 

30 

33 

22 

25 

28 

Reqts  Review  Effort 

Cycle  Time 

Quality 

Email  Routing 

Walkthrough 

Inspections 

Sampling  Inspections 
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Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

1 

4 

7 

7 

10 

13 

18 

20 

22 

8 

10 

12 

1 

2 

3 

1 

4 

7 

1 

5 

9 

2 

3 

4 

25.00% 

40.00% 

55.00% 

50.00% 

55.00% 

60.00% 

80.00% 

85.00% 

90.00% 

65.00% 

70.00% 

75.00% 

Design  Effort 

Cycle  Time 

Quality 

SA/SD 

OOD 

LL 
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UL 

LL 

Avg 

UL 

50 

60 

70 

65 

75 

85 

40 

45 

50 

50 

55 

60 

35 

45 

55 

16 

20 

24 

Design  Review  Effort 

Cycle  Time 

Quality 

Email  Routing 

Walkthrough 

Inspections 

Sampling  Inspections 
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LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

5 

12 

19 

15 

20 

25 

25 

35 

45 

5 

7 

9 

1 

2 

3 

1 

4 

7 

1 

5 

9 

2 

3 

4 

25.00% 

40.00% 

55.00% 

50.00% 

55.00% 

60.00% 

80.00% 

85.00% 

90.00% 

65.00% 

70.00% 

75.00% 

Code  Effort 

Cycle  Time 

Quality 

Manual  w/No  Reuse 

Manual  w/Reuse 

Code  Generation  w/No  Reuse 

Code  Generation  w/Reuse 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

150 

300 

450 

220 

250 

280 

100 

125 

150 

90 

100 

110 

50 

65 

80 

45 

55 

65 

35 

40 

45 

25 

30 

35 

200 

250 

300 

100 

200 

220 

90 

110 

130 

85 

90 

95 
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Scenario  4  (Compose):  “Gestalt”  -  continued 


Option  1  Option  2  Option  3  Option  4 


Code  Review  Effort 

Cycle  Time 

Quality 

Email  Routing 

Walkthrough 

Inspections 

Sampling  Inspections 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

5 

12 

19 

15 

20 

25 

25 

35 

45 

5 

7 

9 

1 

2 

3 

1 

4 

7 

1 

5 

9 

2 

3 

4 

25.00% 

40.00% 

55.00% 

50.00% 

55.00% 

60.00% 

80.00% 

85.00% 

90.00% 

65.00% 

70.00% 

75.00% 

Unit  Test  Effort 

Cycle  Time 

Quality 

Ad  Hoc 

Path  Testing  Only 

Data  Flow  Testing  Only 

Both  Path  and  Data  Flow 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

90 

100 

110 

120 

150 

180 

200 

250 

300 

300 

350 

400 

9 

12 

15 

12 

16 

20 

13 

20 

27 

25 

30 

35 

40.00% 

50.00% 

60.00% 

55.00% 

60.00% 

65.00% 

65.00% 

70.00% 

75.00% 

75.00% 

80.00% 

85.00% 

integration 

Test 

Cycle  Time 

Quality 

Bottom -Up 

Top-Down 

Hybrid 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

55 

60 

65 

40 

50 

60 

35 

40 

45 

20 

25 

30 

20 

25 

30 

20 

25 

30 

55.00% 

60.00% 

65.00% 

55.00% 

60.00% 

65.00% 

70.00% 

75.00% 

80.00% 

SyS‘em  Effort 

Test 

Cycle  Time 

Quality 

On  Breadboard 

On  Brassboard 

Production  Hardware 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

80 

100 

120 

75 

80 

85 

65 

70 

75 

30 

35 

40 

27 

30 

33 

19 

22 

25 

65.00% 

70.00% 

75.00% 

75.00% 

80.00% 

85.00% 

85.00% 

90.00% 

95.00% 

Acceptance  ^ 

_  .  Effort 

Test 

Cycle  Time 

Quality 

Low  Intensity 

Medium  Intensity 

High  Intensity 

LL 

Avg 

UL 

LL 

Avg 

UL 

LL 

Avg 

UL 

15 

20 

25 

25 

30 

35 

50 

60 

75 

3 

5 

7 

8 

10 

12 

15 

25 

35 

70.00% 

75.00% 

80.00% 

80.00% 

85.00% 

90.00% 

90.00% 

95.00% 

99.00% 
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Scenario  4  (Compose):  “Gestalt”  -  continued 


Performa  nee  G  ra  ph 


^jnj x 


"BEST  SOLUTION1 


Frontier 


Values  of  Variables: 

RD  Decision:  1 
RR  Decision:  1 
Design  Decision:  1 
DR  Decision:  1 
Code  Decision:  4 
CR  Decision:  1 
UT  Decision:  1 
IT  Decision:  3 
ST  Decision:  3 
AT  Decision:  1 
Objective:  Overall  Goal:  Mean:  73361 98.7481 1 609 
Requirement  Feasible 


Objective 


ReScale 


Requirement:  TEST01 
Requirement:  TEST02 
Requirement:  TEST03 


0 

0 

0 


This  solution  of  process 
composition  is  optimized  with 

first  priority  of  cycle  time  and 

^secondary  priority  of  quality. 


Additional  details  may  be  found  below. 
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Scenario  4  (Compose):  “Gestalt”  -  continued 


KKKKBEST  SOLUTION” 


Values  of  Variables: 

RD  Decision:  1 
RR  Decision:  4 
Design  Decision:  2 
DR  Decision:  4 
Code  Decision:  4 
CR  Decision:  2 
UT  Decision:  1 
IT  Decision:  3 
ST  Decision:  3 
AT  Decision:  1 
Objective:  Overall  Coal: 
Requirement  Feasible 
Requirement:  TEST01: 
Requirement:  TEST02: 
Requirement:  TEST03: 
Additional  details  may  bi 
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Mean:  012254.401407005 


] 


] 

] 

:  found  below... 


This  solution  of  process 
composition  is  optimized  with 
first  priority  of  quality  and 
secondary  priority  on  cycle  time. 


Scenario  4  (Compose):  “Gestalt”  -  continued 


Subprocesses 

Optimize  for 

Cycle  Time 

Quality 

Requirements  Development 

Traditional 

Traditional 

Requirements  Review 

Email  Routing 

Sampling  Inspections 

Design 

SA/SD 

OOD 

Design  Review 

Email  Routing 

Sampling  Inspections 

Code 

Code  Generation  with  Reuse 

Code  Generation  with  Reuse 

Code  Review 

Email  Routing 

Walkthrough 

Unit  Test 

Ad  Hoc 

Ad  Hoc 

Integration  Test 

Hybrid 

Hybrid 

System  Test 

Production  Hardware 

Production  Hardware 

Acceptance  Test 

Low  Intensity 

Low  Intensity 

Results  (95%  confidence  results  will  not  exceed) 

Cycle  Time 

171 

185 

Quality  Rework  Costs 

$487,000 

$354,000 

Overall  Costs 

$7,935,000 

$841,000 
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QPM  SP  1.3  Select  the  Subprocesses  that  Will 
Be  Statistically  Managed 


Select  the  subprocesses  of  the  project's  defined  process  that  will  be 
statistically  managed. 


j - 1 

I  Select  the  subprocesses  that  we  must  already  measure.  ; 


Subprocesses  that  are  the  major  contributors  to  or  predictors  of  the 
accomplishment  of  the  project’s  interim  or  end-state  objectives  will  be 
selected.  Additionally,  these  need  to  be  suitable  for  statistical 
management.  Statistically  managing  the  selected  subprocesses 
provides  valuable  insight  into  performance  by  helping  the  project 
identify  when  corrective  action  is  needed  to  achieve  its  objectives. 
Select  the  attributes  that  will  measured  and  controlled. 
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Scenario  5:  Deciding  What 
to  Statistically  Manage 


Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  200s  90 

©  2008  Carnegie  Mellon  University 

Scenario  5  (Manage):  “Un-Gestalt” 


We  first  looked  around  to  see  what  data  was  already  being  collected. 

Then  we  discussed  what  additional  data  might  be  easy  to  collect. 

We  wanted  to  ensure  that  the  final  outcomes  of  cost,  schedule  and 
quality  are  measured  so  that  we  can  statistically  manage  these  for 
finished  projects. 

We  have  mixed  feelings!  We  are  collecting  a  lot  of  data  but  not  sure  if 
we  are  using  it  properly.  Sure  hope  it  is  helping  as  it  costs  a  lot  to 
collect  all  of  this  data! 
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Scenario  5  (Manage):  “Gestalt” 


We  began  with  our  leaders  forming  vision  statements  of  our 
organization  over  the  next  2-5  years. 

Then,  we  asked  our  leaders  to  perform  a  “fishbone  diagram”  exercise 
for  each  vision  statement  providing  rich  information  on  barriers  to  each 
vision  statement. 


We  next  asked  our  leaders  to  formulate  a  prioritized  list  of  high  level 
business  goals  attacking  the  barriers  to  the  vision  statements. 


Vision  Stmts 


“Future  State” 


Business  Goal 
Stmts 

“Business  Goals 
tackling  the  Barriers” 
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Scenario  5  (Manage):  “Gestalt”  -  continued 


Once  we  had  our  high  level  business  goals,  we  commenced  on  an 
exercise  called  the  “Goal-Decomposition  Matrix”.  (See  next  slide) 


This  matrix  is  used  to  produce  a  set  of  SMART  Goal  Statements  at  the 
project  level  to  drive  QPM  for  critical  subprocesses. 


Essentially ,  each  project  goal  statement  will  be  a  statement  of  what 
can  be  controlled  at  the  subprocess  level  to  maximize 
accomplishment  of  the  goal. 
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Scenario  5  (Manage):  “Gestalt”  -  continued 


Goal  Decomposition  Matrix 


Process  Step 

Goal 

1 

Goal 

2 

Goal 

3 

Goal 

4 

Goal 

5 

r—  1 

Goal 

6 

•^w  ■ 

Goal 

7 

Req’ts  Elicitation 

X 

X 

p^tacn  a  receives  a^i 
r  S.M.A.R.T.  ' 

objective  statement 
and  is  a  candidate 
for  statistical 

Prototype 

X 

J 

Architecture  Modification 

/ 

High  level  Design 

X 

/ 

Low  level  Design 

X 

management. 

Each  Goal  will 
potentially  have  a 
process 

performance  model 
with  some  of  these 

Coding 

Unit  Test 

Integration  Test 

System  Test 

X 

X 

Alpha  Test 

,  controllable  x  j 

Beta  Test 

X 

ICtl.  LUl  V- 

1 
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Scenario  5  (Manage):  “Gestalt”  -  continued 


Next  year,  we  will  fully  implement  a  Big  Y  -  to  -  small  x  tree  that  is 
connected  with  a  series  of  regression  equations.  With  this  connected 
tree,  we  will  have  a  solid  basis  to  determine  what  to  statistically  manage 
as  well. 

Next  year,  we  will  implement  a  tolerance  analysis  on  our  sub-processes 
to  determine  which  ones  need  to  be  tightly  vs  loosely  controlled. 


Living  the  “High  Life” 

r.  i-  ■  ■  ...  .  i  •  n/r  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

-  Software  Engineering  Institute  CarnegieMellon  March  2008 


©  2008  Carnegie  Mellon  University 


Process-Oriented  Process-Agnostic 


Scenario  5  (Manage):  “Gestalt”  -  continued 


High-Level  Business  Objectives 

(e.g.,  balanced  scorecard) 

Subordinate  Business  Objectives 


y 

y 

y 

y 

y 

y 

y 

y 

\ 

/  \ 

/  x^ 

\ 

x 

/ 

/ 

X 


/ 


\ 


X  \  / 

X  \  / 

X-W- 


T 


\  / 
V  / 


j— X — i  i  Z 


\  '  ' 
'x 


J. 


/ 


N  \  / 


\  /  / 


7 


(e.g.,  $  buckets, 
%  performance) 


High-Level  Processes 


X 

hz 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

J — L 

X 

X 

X 

Subordinate  Processes 

(e.g.,  a  vital 
subprocess  to  be 
statistically  managed) 
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QPM  SP  1.4  Manage  Project  Performance 


Monitor  the  project  to  determine  whether  the  project’s  objectives  for 
quality  and  process  performance  will  be  satisfied,  and  identify 

corrective  apjton  as_apQrpp_riate, _ 

Compare  the  actual  versus  estimated  and  corresponding  actual  trend 
versus  estimated  trend.  If  we’re  not  meeting  our  objectives  or  based 
on  the  actual  trend  it  looks  like  we  won’t  achieve  our  objectives  in  the 
future,  document  what  we  might  do  to  fix  the  shortcoming/potential 
shortcoming. 

Monitor  the  project 

•  Manage  stability  and  capability  of  selected  subprocesses. 

•  Track  quality  and  process  performance  data  including  suppliers’ 

•  Update/calibrate  PPMs  and  predictions  based  on  results  to  date. 

•  Identify  deficiencies/risks  to  achieving  objectives  (e.g.,  where 
current  performance  is  outside  tolerance  intervals,  or 
prediction/confidence  intervals  are  not  contained  within 
specification  limits). 
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Scenario  6:  Periodic 
Management  Reviews  of 
Projects 
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Scenario  6  (Review):  “Un-Gestalt” 


We  hold  many  management  reviews  of  our 
software  measures. 


Sometimes  we  have  management  look  at 
the  control  charts  and  sometimes  they  look 
at  dashboards  that  have  red-yellow-green 
status  codes. 


Our  management  knows  immediately  when 
any  of  our  outcomes  are  unacceptable  or 
go  “out  of  control”. 


However,  our  management  aren’t  sure  if 
they  are  looking  at  the  correct  things  and 
getting  the  value  that  they  should  be! 
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Scenario  6  (Review):  “Gestalt” 


Our  management  mostly  reviews  dashboards 
that  include  not  only  outcomes  but  leading 
indicators  such  as  the  controllable  x  factors 
used  in  our  QPM  and  performance  models. 


We  know  that  just  looking  at  the  outcomes  is 
like  driving  a  car  using  the  rear-view  mirror. 


We  have  also  developed  3-5  leading  indicators 
for  each  outcome  (or  lagging  indicator)  that 
may  be  used  in  a  process  performance  model. 
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Scenario  6  (Review):  “Gestalt”  -  continued 


The  blue  lines 
represent  the 
use  of  process 
performance 
models 
statistically 
predicting 
^outcomes 


Analysis 

indicators 


(leading 

indicators) 
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40 

20 
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Objectives 


>=> 


& 


Strategy  to 
accompli 
tives 


Tasks  to 

accomplish  objectives 


Task  1 
Task  2 
Task  3 

Task  n 


Success 

criteria 


Success  indicators 

(Lagging  Indicators) 


Progress  indicators 

(Lagging  Indicators) 


For  project 
manager 


Actual  " 

Planned 


Reporting  Periods 

Roll-up  for 
higher  management 


Planned 
Reporting  Periods 
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Scenario  6  (Review):  “Gestalt”  -  continued 


Our  management  now  only  spends  20%  of  each  management  review 
looking  at  the  lagging  indictors  (e  g.  the  outcomes  of  cost,  schedule 
and  quality) 

They  now  spend  80%  of  their  time  reviewing  the  statistical 
management  of  controllable  x  factors  and  the  predicted  outcomes 

based  on  the  x  factors. 

Inherently,  the  discussion  focuses  on  management  pro-actively 
taking  action  based  on  performance  models  and  control  charts  of 

controllable  x  factors. 
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QPM  SG  1  Quantitatively  Manage  the  Project 


The  project  is  quantitatively  managed  using  quality  and  process- 
performance  objectives. 

Project  processes  are  managed  against  objectives  using  the 
standard  data  and  statistical  management  spreadsheets*. 

*  Explained  in  QPM  goal  2 

Projects  are  managed  through  the  use  of: 

•  measuring  and  controlling  quality  and  process  performance 
attributes. 

•  statistical  techniques  to  ensure  stable  and  capable  subprocesses 

•  PPMs  to  predict  if  objectives  will  be  met  based  on  current 
performance 

•  spec  limits  to  indicate  when  the  performance  of  current  processes 
will  adversely  affect  the  project’s  ability  to  meet  its  objectives 
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QPM  SG  2  Statistically  Manage  Subprocess 
Performance 


The  performance  of  selected  subprocesses  within  the  project's  defined 
process  is  statistically  managed. 
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QPM  SP  2.1  Select  Measures  and  Analytic 
Techniques 


Select  the  measures  and  analytic  techniques  to  be  used  in  statistically 
managing  the  selected  subprocesses. 

Select  effort,  size,  and  defects  (estimated  and  actual  for  each)  and 
use  trend  charts  to  analyze  them  and  investigate  spikes  that  appear 
to  be  unusually  large  as  special  causes. 


Identify  the  measures  that  will  provide  insight  into  the  performance  of 
the  subprocesses  selected  for  statistical  management  and  the 
statistical  techniques  that  will  be  used  for  analysis.  These  measures 
can  be  for  both  controllable  and  uncontrollable  factors.  Operational 
definitions  will  be  created/updated  for  these  measures.  Where 
appropriate  (i.e.,  they  are  critical  to  meeting  downstream  objectives), 
spec  limits  will  be  established  for  the  measures. 
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QPM  SP  2.2  Apply  Statistical  Methods  to 
Understand  Variation 


Establish  and  maintain  an  understanding  of  the  variation  of  the 
selected  subprocesses  using  the  selected  measures  and  analytic 
techniques. 

For  each  subprocess  measure,  compare  the  actual  to  the  estimated 
(using  trends)  to  understand  how  much  variation  there  is  between 
what  we  expected  and  what  we  are  actually  getting. 


Selected  measures  for  the  subprocesses  will  be  statistically 
controlled  to  identify,  remove,  and  prevent  reoccurrence  of  special 
causes  of  variation,  or  in  other  words,  stabilize  the  process.  When 
control  limits  are  too  wide,  sources  of  variation  are  easily  masked  and 
further  investigation  is  warranted. _ 
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QPM  SP  2.3  Monitor  Performance  of  the 
Selected  Subprocesses 


Monitor  the  performance  of  the  selected  subprocesses  to  determine 
their  capability  to  satisfy  their  quality  and  process-performance 
objectives,  and  identify  corrective  action  as  necessary. 


L 


Compare  the  actual  versus  estimated  and  corresponding  actual  trend 
versus  estimated  trend.  If  we’re  not  meeting  our  objectives  or  based 
on  the  actual  trend  it  looks  like  we  won’t  achieve  our  objectives  in  the 
future,  document  what  we  might  do  to  fix  the  shortcoming/potential 
shortcoming. 


j 


For  a  stable  subprocess,  determine  if  the  control  limits  (natural 
bounds)  are  within  the  specification  limits  which  indicates  a  capable 
subprocess.  If  it  is  not,  document  corrective  actions  that  address  the 
capability  deficiencies. 
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QPM  SP  2.4  Record  Statistical  Management 
Data 


Record  statistical  and  quality  management  data  in  the  organization’s 
measurement  repository. 

!  Put  the  data  in  our  statistical  management  spreadsheet. 


Record  the  data  along  with  sufficient  information  to  understand  the 
context  for  the  data  and  thus  make  the  data  usable  by  the 
organization  and  other  projects. 
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QPM  SG  2  Statistically  Manage  Subprocess 
Performance 


The  performance  of  selected  subprocesses  within  the  project's  defined 
process  is  statistically  managed. 

r - | 

!  Systemization  of  our  process  is  achieved  through  planning  and  \ 

!  execution  of  the  plans.  I 

l _ j 


Selected  subprocesses  are  statistically  managed  to  ensure  stability 
and  capability  (i.e.,  special  causes  of  variation  are  identified, 
removed,  and  prevented  from  recurring  and  the  control  limits  of  the 
subprocess  are  kept  within  the  specification  limits). 
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CAR  SG  1  Determine  Causes  of  Defects 


Root  causes  of  defects  and  other  problems  are  systematically 
determined. 
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CAR  SP  1.1  Select  Defect  Data  for  Analysis 


Select  the  defects  and  other  problems  for  analysis. 
|  Select  first  ten  defects/problems  on  the  list 


Defects  and  other  problems  are  selected  for  further  analysis  based 
on  factors  such  as  clustering  and  analysis  of  the  clusters  of  similar 
defects  or  problems  including  impact  to  the  project’s  objectives, 
predicted  ROI,  etc.  PPMs  may  be  used  in  the  prediction  of  impact, 
calculation  of  costand  benefits,  ROI,  etc. 
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CAR  SP  1.2  Analyze  Causes 


Perform  causal  analysis  of  selected  defects  and  other  problems  and 
propose  actions  to  address  them. 

Perform  causal  analyses  on  the  selected  defects  and  problems  using 
Fishbone  diagrams.  The  analysis  is  qualitatively  driven.  Propose 
actions  to  address  the  identified  causes. 


The  causal  analysis  can  include: 

•  analysis  of  PPBs  and  PPMs  to  help  identify  potential  sources  of 
defects  and  problems 

•  causal  analysis  meetings  with  the  involved  parties 

•  formal  root  cause  analysis. 

The  analysis  is  both  quantitative  and  qualitative. 

Actions  are  proposed  to  not  only  address  the  defect/problem  but  also 
to  correct  the  root  cause  and  prevent  reoccurrence. 
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CAR  SG  1  Determine  Causes  of  Defects 


Root  causes  of  defects  and  other  problems  are  systematically 
determined. 

r - —  — - 

I  Systemization  of  our  process  is  achieved  through  planning  and 

!  execution  of  the  plans. 


Processes,  plans  and  methods  are  used  to  identify  the  root  cause(s) 
of  defects  and  other  problems  and  identify  the  actions  necessary  to 
fix  and  prevent  future  occurrences. 
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CAR  SG  2  Address  Causes  of  Defects 


Root  causes  of  defects  and  other  problems  are  systematically 
addressed  to  prevent  their  future  occurrence. 
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CAR  SP  2.1  Implement  the  Action  Proposals 


Implement  the  selected  action  proposals  that  were  developed  in  causal 
analysis. 

l  l 

■  Execute  proposed  actions.  ; 


Prioritize  the  actions  based  on  factors  such  as  impact,  ROI, 
availability  of  resources/budget,  interdependencies,  etc.  Implement 
the  actions.  Additionally,  identify  and  remove  similar  defects  and 
other  problems  that  may  exist  in  other  processes  and  work  products. 
Where  appropriate,  submit  proposals  to  improve  the  OSSP. 
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CAR  SP  2.2  Evaluate  the  Effect  of  Changes 


Evaluate  the  effect  of  changes  on  process  performance. 
r - — - — - 

I  Did  process  performance  go  up/down  (e.g.,  more/less  productivity, 
j-  less/more  defects). 


Measure  and  analyze  the  change  to  determine  if  process 
performance  has  been  positively  affected  and  there  are  no  harmful 
side-effects.  This  may  involve  hypothesis  testing  using  a  before  and 
after  PPBs  to  determine  if  the  change  is  statistically  significant.  May 
also  involve  comparing  the  change  to  the  PPM  predicted  change  to 
see  if  the  predicted  performance  benefits  were  achieved.  Further 
analysis  may  use  a  PPM  to  determine  if  the  change  will  positively 
contribute  to  meeting  downstream  quality  and  process  performance 
objectives. 
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CAR  SP  2.3  Record  Data 


Record  causal  analysis  and  resolution  data  for  use  across  the  project 
and  organization. 

|  Put  the  data  in  our  spreadsheet. 


Record  the  data  along  with  sufficient  information  to  understand  the 
context  for  the  data.  Data  related  to  project  adoption  experience  and 
other  data  that  will  assist  deployment  in  other  parts  of  the 
organization  should  be  collected. 
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CAR  SG  2  Address  Causes  of  Defects 


Root  causes  of  defects  and  other  problems  are  systematically 
addressed  to  prevent  their  future  occurrence. 

T“ - ‘ - - - “  “ - — - -  -  - - — - “  “ - -  - - -  - 

!  Systemization  of  our  process  is  achieved  through  planning  and 
^execution  of  the  plans. 


The  changes  are  made  and  measures  taken  and  analyzed  to 
determine  if  the  changes  are  positive  and  statistically  significant. 
Similar  processes  and  work  products  are  also  modified  and  sufficient 
data  is  recorded  to  understand  the  context  and  assist  other  projects. 
When  appropriate,  proposals  are  submitted  to  the  organization  to 
improve  the  OSSP. _ 
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Scenario  7:  Taking 
Corrective  Action  When 
Needed 


Living  the  “High  Life” 

,^=-  Software  Engineering  Institute 

•  m«-  11  Rusty  Young,  Bob  Stoddard,  Mike  Konrad 

Carnegie  Mellon  March  200s  119 

©  2008  Carnegie  Mellon  University 

Scenario  7  (CAR):  “Un-Gestalt” 


Our  projects  use  pareto  analysis  and  fishbone  diagrams  to  decide 
which  problems  are  the  greatest  importance  to  tackle. 

We  work  very  hard  to  resolve  all  defects  and  process  issues.  There 
are  so  many  of  them,  that  we  seem  to  be  expending  all  of  our  time 
resolving  defects  and  issues. 

With  the  volume  that  we  have,  we  have  now  decided  to  staff  more 
engineers  throughout  the  project’s  lifecycle  to  handle  the  workload. 
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Scenario  7  (CAR):  “Gestalt” 


Our  project  uses  a  closed-loop  corrective  action  process  similar  to  the 
Ford  Global  8D  process.  We  have  modified  the  process  to  make 
specific  uses  of  process  performance  baselines  and  models  at  the 
points  indicated: 
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Scenario  7  (CAR):  “Gestalt” 


Our  project  uses  a  closed-loop  corrective  action  process  similar  to  the 
Ford  Global  8D  process.  We  have  modified  the  process  to  make 
specific  uses  of  process  performance  baselines  and  models  at  the 
points  indicated: 
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Scenario  7  (CAR):  “Gestalt” 


Our  project  uses  a  closed-loop  corrective  action  process  similar  to  the 
Ford  Global  8D  process.  We  have  modified  the  process  to  make 
specific  uses  of  process  performance  baselines  and  models  at  the 
points  indicated: 
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Scenario  7  (CAR):  “Gestalt” 


Our  project  uses  a  closed-loop  corrective  action  process  similar  to  the 
Ford  Global  8D  process.  We  have  modified  the  process  to  make 
specific  uses  of  process  performance  baselines  and  models  at  the 
points  indicated: 


Scenario  7  (CAR):  “Gestalt” 


Our  project  uses  a  closed-loop  corrective  action  process  similar  to  the 
Ford  Global  8D  process.  We  have  modified  the  process  to  make 
specific  uses  of  process  performance  baselines  and  models  at  the 
points  indicated: 


pr 


We  use  our  PPBs  and 
PPMs  to  predict  the 
impact,  upon 
deployment,  of  the  new 
solution  and  compare 
to  actual  impact. 


>engage  CAteam 
ifter  recognition 
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OID  SG  1  Select  Improvements 


Process  and  technology  improvements,  which  contribute  to  meeting 
quality  and  process-performance  objectives,  are  selected. 
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OID  SP  1.1  Collect  and  Analyze  Improvement 
Proposals 


Collect  and  analyze  process-  and  technology-improvement  proposals 

|  Put  the  process  and  technology  improvement  proposals  in  a 
;  spreadsheet,  think  about  each  one,  and  tag  with  a  plus  if  you  think  it 
I  will  improve  or  a  minus  if  you  think  it  will  decrease  quality  and 
!  process  performance. 


Collect  improvement  proposals  and  analyze  for  costs,  benefits,  and 
risks.  Select  those  that  will  be  piloted.  Document  the  results  of 
analyses  and  selection.  PPMs  may  be  used  to  predict  effects  of  the 
change  to  the  process,  the  potential  benefits,  evaluate  side  effects, 
and  evaluate  the  effects  of  multiple  interrelated  improvement 
proposals. 
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OID  SP  1.2  Identify  and  Analyze  Innovations 


Identify  and  analyze  innovative  improvements  that  could  increase  the 
organization’s  quality  and  process  performance. 

r - - 

I  Identify  improvements  that  seem  to  be  “out  of  the  box”  and  look  like 
Uhey  will  increase  quality  and  process  performance. 


Actively  seek,  both  inside  and  outside  the  organization,  innovations  to 
improve  processes  and  product  technologies  and  analyze  them  for 
possible  inclusion,  predicting  cost  on  benefits  (using  PPMs).  Use 
PPMs  and  PPBs  to  analyze  the  OSSP  and  identify  areas  or  targets  of 
opportunity  for  change.  Submit  improvement  proposals  for  changes 
that  are  predicted  to  be  beneficial.  Select  those  to  be  piloted. 
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OID  SP  1.3  Pilot  Improvements 


Pilot  process  and  technology  improvements  to  select  which  ones  to 
implement. 


i 

L 


Try  the  improvements  or  use  someone  else’s  results  and  see  which 
ones  might  be  selected. 


1 


Plan  the  pilot  including  documenting  the  criteria  for  evaluating  the 
success  or  failure  of  the  pilot.  Select  pilot  environments  that  are 
representative  of  the  typical  use  of  the  improved  process  and/or 
technology.  Evaluate  the  results  using  the  documented  criteria.  This 
will  typically  involve  the  use  of  PPMs  to  see  if  the  processes  behaved 
as  predicted  and  PPBs  to  see  it  the  change  is  statistically  significant 
(through  the  use  of  hypothesis  testing). 
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OID  SP  1.4  Select  Improvements  for 
Deployment 


Select  process  and  technology  improvements  for  deployment  across 
the  organization. 

|  Pick  the  improvements  to  be  deployed  across  the  organization. 


Prioritize  the  improvements  for  deployment  (typically  involves 
evaluating  the  predicted  ROI  from  PPMs  and  other  factors  such  as 
availability  of  resources,  impact,  etc.)  and  begin  to  determine  a 
deployment  strategy. 
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OID  SG  1  Select  Improvements 


Process  and  technology  improvements,  which  contribute  to  meeting 
quality  and  process-performance  objectives,  are  selected. 

|  Improvements  that  appear  to  help  us  meet  our  goals  are  selected. 


The  improvements  which  will  contribute  most  to  achieving  the 
organizations  objectives,  provide  the  best  ROI  and  most  desirable 
impact,  and  can  be  accomplished  with  available  resources  will  be 
chosen. 
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OID  SG  2  Deploy  Improvements 


Measurable  improvements  to  the  organization's  processes  and 
technologies  are  continually  and  systematically  deployed. 
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OID  SP  2.1  Plan  the  Deployment 


Establish  and  maintain  the  plans  for  deploying  the  selected  process 
and  technology  improvements. 

r  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  — 

!  Schedule  the  deployment  of  the  improvements  and  update  the 
■  schedule  as  necessary. 


Determine  modifications  necessary  for  deploying  the  new/revised 
process  to  the  projects’  environments.  Define  how  the  value  of  the 
deployed  process/technology  improvements  will  be  measured. 
Determine  the  deployment  risks.  Devise  a  plan  for  the  deployment, 
get  commitment  from  stakeholders,  and  revise  as  necessary. 
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OID  SP  2.2  Manage  the  Deployment 


Manage  the  deployment  of  the  selected  process  and  technology 
improvements. 

|  Track  against  the  schedule  and  reschedule  as  necessary. 


Monitor  the  deployment  against  the  plan  and  determine  that  the 
deployed  processes  have  not  adversely  affected  the  ability  to  meet 
quality  and  process  performance  objectives.  Update  the  appropriate 
PPMs  and  PPBs. 
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OID  SP  2.3  Measure  Improvement  Effects 


Measure  the  effects  of  the  deployed  process  and  technology 
improvements. 

■^Measure  whether  people  like  the  change. 


Measure  the  cost  and  value  of  the  improvement  in  the  deployed 
process.  Through  the  use  of  PPMs  determine  if  the  predicted 
performance  is  being  achieved.  Use  hypothesis  testing  or  other 
statistical/probablistic  techniques  of  the  before  and  after  PPBs  to 
determine  if  the  improvement  is  statistically  significant. 
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OID  SG  2  Deploy  Improvements 


Measurable  improvements  to  the  organization's  processes  and 
technologies  are  continually  and  systematically  deployed. 

p “ —  —  —  —  —  ———————————————————————  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  —  — 

I  Measured  improvements  that  help  are  adopted  according  to  our 
^approved  plans. 


We  have  ensured  through  measurements  and  analyses  that  the 
deployed  processes  have  indeed  been  systematically  and  continually 
improved  the  process  in  a  statistically  significant  way. 
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Scenario  8:  Introducing 
Innovative  Change  to 
Organization 
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Scenario  8  (OID):  “Un-Gestalt” 


Our  organization  benchmarks  with  other  companies  to  stay  informed  of 
the  leading-edge,  innovative  concepts 


Based  on  word  of  mouth  and  expert  opinion,  we  identify  the  low 
hanging  fruit  new  concepts  to  try  out  each  year. 


We  pilot  all  of  the  new  concepts  each  year  that  we  can  afford  to. 


Hopefully,  this  will  pay  off.  It  does  represent  a  lot  of  time  and 
resources. 
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Scenario  8  (OID):  “Gestalt” 


Our  organization  possesses  a  healthy  collection  of  process 
performance  baselines  and  models  developed  over  a  multi-year  period. 


With  such  an  arsenal,  we  are  able  to  use  them  to  first  look  inward  and 
identify  the  ripe  opportunities  for  radical  improvement  and  innovation. 


Once  we  identify  the  areas  ripe  for  improvement,  we  benchmark  with 
external  organizations  for  the  types  of  innovation  we  need. 
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Scenario  8  (OID):  “Gestalt”  -  continued 


As  we  identify  external  innovation  ideas,  we  use  our  baselines  and  \ 
models  to  evaluate  the  potential  of  the  ideas.  In  this  manner,  we  will 
use  our  baselines  and  models  to  “screen”  the  ideas  to  pilot. 


Once  we  identify  the  ideas  to  pilot,  we  use  the  baselines  and  models  to 
predict  the  outcomes  we  should  see. 


We  then  pilot  and  compare  the  results  to  our  prediction.  We  make 
adjustments  as  necessary  before  rollout. 


Then  we  rollout  and  use  baselines  and  models  to  track  the  newf  4 
subprocess  changes  during  adoption  to  steady  state  running.  V_ 
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SOME  FINAL  THOUGHTS 
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HM  Involves  and  Impacts  the  Entire  Organization 


Manufacturing 


Contracts 


Customer/ 
End  User 


Other 
Projects 


Project 


Sales/ 

Marketing 


Training 


HR 


=  Software  Engineering  Institute  CarnegieMellon 


Living  the  “High  Life” 

Rusty  Young,  Bob  Stoddard,  Mike  Konrad 
March  2008 

©  2008  Carnegie  Mellon  University 


Are  you  just  in  it  for  the  number? 


That  can  be  a  valid  business  objective 

But,  it  is  in  all  of  our  best  interest  to  ensure  that  the  number  means 
something 

•  That  means  paying  attention  to  the  informative 

•  The  richness  of  the  model  is  in  the  informative 

•  The  ideas/concepts  that  add  value  are  in  the  informative 

Without  the  informative  material  Levels  4  and  5  add  little  of  even  the 
minimum  we  all  believe  they  are 

If  it  is  not  value  added,  change  it 
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Lack  of  Data  is  No  Excuse 


In  fact,  it  is  quite  common 
And  the  answer  is 


Sampling 
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Can  Level  5  be  Stagnant? 


Can  performance  and  quality  improvement  be  characterized  as 
asymptotic? 

Since  every  one  loves  “how  many”  questions 

•  How  many  “improvements”  must  be  made  to  get  to  and  remain 
at  level  5? 
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